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ABSTBACT 

This booklet identifies basic techniques and 
methodologies for evaluating training and development programs* It 
.also examines a selected number of methodologies that appear to be 
applicable to evaluating the results of program performance and 
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general information on evaluation and revievs some basic assumptions 
about evaluative research. ^ Section 2 gives a brief explanation of the 
process of evaluation* Section 3 summarizes several selected program 
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1.0 INTRODUCTION 



During the last two decades there has been increasing skepticism regarding the 
effectiveness of many federally- funded social and economic programs. Mixed feelings 
concerning both conventional and the more innovative techniques and approaches have 
jLiitroduced the need for means through which the success or failure of programs can be 
appropriately assessed. Questions have been raised about the techniques for deter- 
mining program performance and achievement and the relevancy of organized programs in 
, many areas ^ including community action^ urban renewal^ healthy education^ criminal jus- 
tice^ employment^ transportation ^ and social welfare. 

This volvmve seeks to identify the more basic techniques and methodologies for 
evaluating training and development programs. It is anticipated that such an identifi- 
cation will assist program administrators ^ evaluators^ policy-makers^ program staff 
persons r and citizens in understanding more fully existing program evaluation techniques. 

"Interest in evaluation research/' says Caro^ **has been greatly stimulated in 
the past decade by widespread concern for domestic social reform. In an atmosphere 
^ charged with demands for rapid and significant change^ a great many innovative action 
program;" have been introduced."^ The emergence of programs designed to bring about 
social reform has been accompanied by demands for strategic kinds of changes in our 
institutional service delivery systems. Various strategies have been utilized in plan*- 
ning the more innovative and experimental program designs. This handbook examines a 
selected nximber of methodologies which appear to be applicable — with some modifications ^ 
perhaps — to evaluating the results of program performance and achievement in many 
social and economic programs. 

Related materials about evaluation research designs have been summarized into 
several mode?.::; for measuring program results. While the information given is not com- 
prehensive in scope f it does offer some guidance and direction to those concerned with 
developing evaluation designs. Applicable concepts and methods for measurement are 
also presented. Finally^ the more basic intent of this voluuue is to docuuuent what is 
perceived to be relevant ideas and findings on social measurement techniques ^ and bib- 
liographic sources. 

This guide for evaluation includes five sections. Section I provides general 
information on evaluation and reviews some basic assuunptions about evaluative research. 
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Section II gives a brief explanation of the process of 'evaluation. Selected program 
evaluation models have been included in Section III, with emphasis on a comparison 
of model and the simplification of evaluation designs. The final two sections — IV 
and V — reviews other approaches to evaluation and contain bibliographic references 
and related materials , respectively. 
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Section One 



General Information 
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2.0 GENERAL INFORMATION 



The need for objectively evaluating programs extends beyond the simple demand 
for systematic appraisal. One of the major contributions of evaluating and assessing 
the effectiveness of programs lies in an analysis of the program^s worth and its 
objectives, the assumptions xinderlying those objectives, the rational!;- for believing 
that these activities are capable of attaining the objectives. 

Several fundamental assumptions underlie the need for ^.valuating programs and 
activities: 2 

1) Any training or development program must validated; that is, the ef- 
ficiency and effectiveness of programs must be objectively determined. Programs must 
be subjected to critical evaluation and must demonstrate their worth if they are to be 
continued. 

2) Any training or development program can be improved. This is necessary 
even if the effectiveness of the program has been established, innovative approaches 

may be developed, alternative courses of action may be delineated — providing further O 
refinements to the program's effectiveness. 

3) Program improvements can be effectuated in the following manner: 

a) Objective and coordinated evaluation of every aspect of the operation. 

b) The application of imagination and creative thinking by program personnel. 

c) Deliberate collection of the observation, ideas, and thinking of all 
program personnel. 

d) Critical analysis and synthesis of findings, alternatives, and ideas. 

e) Systematic, time-phased development and tryout of policies and procedures 
as well as identification of resources (people, equipment, materials, 
time. Space, and money) needed to carry out plans. 

It follows from the above assumptions that certain principles must guide the 
evaluation effort. ''To be effective, evaluation of training and deveiopment programs 
must be conducted in such a way that they are consistent with the purposes, objectives 
and goals and are in accordsuice with proven principles of evaluation. These are the 
principles which should guide all evaluation efforts: 

i 
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1) Evaluation must conducted in terms of purposes * A critically important 
aspect of this pirincxple is that the purposes of the evaluation must be 
clearly xinderstood by all concerned* 

2) Evaluation must be cooperative * All who are a part of the process of ap- 
praisal or who will be affected by it should participate in the process* 
In brief/ total involvement is essential to the success of the project* 

3) Evaluation roust be continuous * On-going appraisals should be conducted 
with appropriate analysis # follow-up , and re-appraisal* The form, en5)hasis, 
and focus may shift, but the evaluation process must be one that never 
stops a continuous process* 

4) Evaluation must be specific * Specificity is the key to ijuccessful evalua-- 
ti on* To the extent that evaluation pinpoints strengths and weaknesses # 
it can serve the purpose for which it was intended* Generalizations con- 
tribute nothing to an improvement or demonstration program* 

5) Evaluation must provide a means for appraisal * Inherent in any form of 
evaluation must be the means and fccus for personnel to be able to appraise 
themselves/ their practices, and their products* This holds true for 
social agencies* government, business and industry, educational institiu*" 
tions, and other institutions* 

6) Evaluation must be based on a uniform and objective methodology * Prior 
to an attempt at evaluation should be the estat>iisnment of standards and 
basic criteria* These standards and criteria must be acceptable, readily 
applicable and observable in product and process* Uniformity in procedure 
and in standards for measurement insures comparable findings and conclu- 
sions* 



2*1 The Evaluation Process 



The purpose of evaluation yesearch is to provide information for deci si on -makers 
about programs* is not a single-step process* Instead, it einbodies a body of con- 
cepts which are applied in such a way that they can contribute to the improvement of 
programs and practical activities* 

Suchman/ in defining evaluation, provides key conceptual elements in a defini- 
tion of evaluation from a methodological point of view* These include the following: 
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1) a planned program of deliberate intervention ^ not just any natural or 'accidental* 
event; 2) an objective or goal which is considered desirable or has some positive value ^ 
not Bimply whatever change occurs; and, 3) a method for determining the degree to which 
the planned program achieves the desired objectives. Evaluative research asks about 
the kind ofdiange desired, the means by which this change is to be brought about, and 
the signs according to which such change can be recognized.^ He further advises that 
an evaluation study should extend beyond the administrative goal of assigning a passing 
or failing grade to the program, it is believed that it should attempt to find out 
why a program was or was not effective (a research goal). To provide such ani^y^ers 
requires an analysis of such factors as 1) the attributes of the program itself that 
make it more or less successful; 2) the population exposed to the program in terms of 
which subgroupd are reached and which affected; 3) the situational context within 
which the program takes place, such as auspices, locale, competing programs, and pv\blio 
opinion; and 4) the different kinds of effects produced by the program, such as cog- 
nitive, atritudinal or behavioral, long or short term, unitary or multiple. Special 
attention should be given to negative side-effects, in this instance evaluation would 
entail more judging and in-depth research on conditions affecting success or failure. 

The different viewpoints of authorities in the field reflect concern for in- 
formation relative to program outcome and the value of the programs. Caro reviews 
rather thoroughly literature on evaluation and how the concept is defined by many 
authors. Summarizincr what is c ontained in the overview of evaluation research, the 
viewpoints may be d ^ tized along several lines of inquiry. Some authorii emphasize 
the data collectior ^ct of evaluation; others, identify categories for evaluation; 
and still others d^ on the judgmental dimensions of the evaluation process. Greenberg, 
Brooks/ and Suchman, for instance, refer to evaluation as the ''procedure by which pro- 
grams are studied to ascertain their effectiveness in the fulfillment of goals. "4 They 
also en5)hasize the need to measure the relative impact of key program variables, and 
an attempt is made to identify essential categories for evaluation. Methodological 
considerations are also noted in a discussion of the judgmental dimension of evaluation. 
This dimenaion is viewed as one which combines performance data with a goal scale. 
Attempts are also made to assess worth or social utility, in the latter instance, the 
reader may find that evaluation is a process of making judgements relative to a pro- 
gram"* s worth/ with emphasis on both input and output variables so that the achievement 
of program goals can be properly determined. 

The next several pages will be devoted to a review of the process of evaluation, 
program evaluation models, other approaches to evaluation, the measurement and proces- 
sing of data used, and an overview of jnajor steps in setting up a general evaluation 
of prograras. 
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The Evaluation Process 



3 * 0 THE PROCESS OF EVALUATION 



Evaluation research is designed to provide answers to policymakers concerning 
decisions relative to the c ontinuation of a program, extensions of activities and loca- 
tions, modification of projects, or termination of programs, if it is found that they 
are in^^f fective* 

Traditionally, evaluation research has consisted of five basic stages: 

Stage 1 * Goal assessment — Finding out the goals of the program* 

Stage 2 * Goal achievement ™ Translating the goals 'into measurable indica- 
tors of goal achievement* ^ 

Stage 3 * E>Tternal measurement Collect^ing data on the indicators for those 
who h ave not been exposed to the programs* ; 

Stage 4 * Internal measurement ™ Collecting similar data on an equivalent 
group that has been exposed to the program* 

Stage 5 * Comparing the data on program participants and controls in terms 
of goal criteria* 

In most evaluations f similar problem-solving procedures are used* Tracey lists 
several steps in problem-solving in his discussion on evaluating training and develop- ^ 
ment systems;^ 

First f the need for evaluation is recognized, the areas to be 
evaluated or measured are identified^ and the procedures and 
instruments to be used in the evaluation are selected or de-^ 
veloped* Then evaluators are chosen and trained in the pro- 
cedures and in the use of the instruments* Once the data are 
collected and analyzed, conclusions are drawn and alternative 
courses of action are identified* Finally, the decision or 
course of action is subjected to trial, and the results are 
checked* 

In general, these stages in the evaluative process will suffice in determining program 
successes and failures* It should be understood, however, that these are some pervasive 
problems associated with evaluation designs* 
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Using an educational program as an example^ the audit may involve all training 
and' development subsystems* Basically ^ the subsystems for educational program would 
likely Include the following: 

3*2*1* Philosophy and Goals * The object of scrutiny in this regard would be 
an exploration of basic beliefs about training and development that 
guide the plans and operations of activity ^ and how these beliefs re- 
late to the stated goals of the organization* 

3*2*2* Buildings and Facilities * An examination of buildings^ Srpace^ equip- 
mentr and facilities are made for determining the adequacy and ap" 
proprlateness of these facilities* 

3*2*3* Management * The quality of the planning^ organizing^ direction^ and 
control of the activity is evaluated* 

3*2*4* Staff and Faculty * An audit Is made of the qualification and performance 

of the implementers of educational training and policies* ^ 

3*2*5* Curriculum* The substarttee of the training program (the subject matter 
to be taught and learned?* is analyzed for effectiveness* Basic in*- 
structlonal strategies are also assessed^ including training methodolo-- 
gy^ medla^ and systems of tralnee-lnstructor organization needed to 
conduct training* 

3*2*6* Instructional Support * An analysis is made of Instructional support 
activities such as ^instructors training programs^ training literature 
and publications ^ guids^nce and counseling services ^-^ the adequacy of 
each jtype of support* 

Evaluation designs for each of the trailing and development subsystems have been il^ 
lustrated by Tracey* Some of the example? are shown in Figure 1* 
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Using an educational program as an example^ the audit may involve all training 
and' development subsystems. Basically^ the subsystems for educational program would 
likely include the following: 

Philosophy and Goals . The object of scrutiny in this regard would be 
an exploration of basic beliefs about training and development that 
guide the plans and operations of activity ^ and how these beliefs re-^ 
late to the stated goals of the organization. 



3 . 2.1. 

3.2. 2 . 

3 . 2£ . 3 . 
3.2.4. 
3 . 2£ . ^ . 



Buildings and Facilities . An examination of buildings^ ?pace^ equip- 
mentr and facilities eore made for determining the adequacy and ap- 
propriateness of these facilities. 

Management . The quality of the planning^ organizing^ direction^ and 
control of the activity is evaluated. 

Staff and Faculty . An audit is made of the qualification and performance 
of the implementers of educational training and policies. 

Curricul um. The substarite of the training program (the subject matter 
to be taught and learnedr is analyzed for effectiveness. Basic in*- 
structional strategies are also assessed^ including training methodolo-^ 
gy^ media ^ and systems of trainees-instructor organization needed to 
conduct training. 



3.2£.6. 



Instructional Support . An analysis is made of instructional support 
activities such as ^instructor ^ training programs^ training literatxire 
and publications, guidance and counseling services — the adequacy of 
each jtype of suppott. 



Evaluation designs for each of the trailing and development subsystems have been il-^ 
lustrated by Tracey. some of the exemiples are shown in Figure 1. 



9 



Figure 1 : 

lUiUSTRATION'^F^AN EVALUATION DESIGN FOR TRAINING AND DEVELOPMENT 
PROGRAMS (Principles and Criteria)* 



BASIC COliPONENT 



CRITERIA FOR MEASUREMENT 



RATING STANDARDS 



EVALUATION COMMENTS 



I. Purpose and 
Philosophy 

-purpose 

-philosophy 

-objectives 



A clear statement of pur- 
pose and role in the edu- 
cational and business 
world. An institution's 
integrity is measured not 
only in terms of its 
stated purpose but also 
in terms of its genuine 
efforts to fulfill that 
purpose . 

A well-developed and ap- 
propriate statement for 
its specific role. 



1 . Satisfactory 

2. Needs Improve- 
ment 

3 . Unsatisfactory 



II . Organization 
and Adminis-" 
tration 

-^Organiza- 
tional 
chart 

-By-laws and 
personnel 
manuals 



Coordinating various re- 
sources to accomplish the 
objectives of the agency 
and institution. 

There should be a clear 
differentiation between 
the policy-making func* 
tions of governing boards 
or committees and those 
in charge of* administer- 
ing policies. 



1 . Satisfactory 

2. Needs Improve- 
ment 

3 . Unsatisfactory 
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ILLUSTRATION OF AN EVALUA3?I0N DESIGN FOR TRAINING AND DEVELOPMENT 
PROGRAMS {Principles and Criteria)* 



BASIC COMPONENT 


1 

CRITERIA FOR MEASUREMENT 


RATING STANDARDS 


EVALUATION COMMENTS 


^Expressed evi- 
dence of inde- 
pendence 

■ 








^l I • Educational Pro- 
gram 

"^Admission 

--Curriculum 

-^Instruction 




The program must be clear- 
ly related to the purposes 
of the institution* This 
must be demonstrated in 
policies on admission ^ 
content of curricula ^ grad- 
uation requirements^ in- 
structional methods and 
procedures r and quality 
of work required* 


1 * Satisfactory 
2* Needs Improve- 
ment 

3* Unsatisfactory 




XV* Financial 
Resources 

-Sources of In-^ 

cotne 
-^Stability of 

Income 
-^Or gani z a t ion 
for proper 
admin ist rat ion 

-•Educational 
Expenditures 

-^Budget prepa** 
ration 

-^Budget control 

'Relationship 
to External 
Budget control 


Financial resources of an 
institution is to be 
judged in relation to the 
basic, purposes of l:he 
the institution * 

The organization of the 
business structure and 
the financial control 
of fin€mcial resources 
should reflect th'^j fact 
that financial re- 
sources are tools of the 
enterprise* 

The institution should 
have soxind budgeting 
and controls proper 
records ^ reporting ^ 


1* Satisfactory , 
2* Needs Improve- 
ments 
3* Unsatisfactory 
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ILLUSTRATION OF AN EVALUATION DESIGN FOR TRAINING AND DEVELOPMENT 
PROGRAMS (Principles and Criteria) 



BASIC COMPONENT 



CRITERIA FOR MEASUREMENT 



RATING STANDARDS 



EVALUATION COMMENTS 



-Accounting , 
Reporting, 
and auditing 

-I^lanagement of 
Income 

-Purchasing & 
Control 



and auditing. 



V* Personnel 

-Recruitment 
and selec- 
tion 
-Personnel org- 
anization 
-Training and 

preparation 
•"Professional 

growth 
^Financial 

security 
-Tenure and 
academic f 
freedom 
-Criteria and 
procedure for 
evaluation 
-Promotions $t 
Salaries 



The selection r develop- 
ment , and retention of 
competent personnel at 
all levels is of major 
importance . 

The relationship between 
personnel objectives and 
institutional purposes 
determines in large 
measure the effective- 
ness of programming* 

There should be person- 
nel participation in the 
development of institu- 
tional policies. 

The performance of per- 
sonnel determines the 
quality of the insti- 
tution* 



1* Satisfactory 
2* Needs Improve- 
ment 

3* Unsatisfactory 
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IIJ>USTRATION OF AN EVALUATION DESIGN FOR TRAINING AND DEVELOPMENT 
PROGRAMS (Principles and Criteria) 



BASIC COMPONENTS 



CRITERIA FOR MEASUREMENT 



RATING STANDARDS 



VI. Management 
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-Objectives 
-Long-range and 

short-range 

plans 

-Policy manual 

•■Written policies 

-Procedure man-- 
ual 

--Schedules 

-Special Programs 

-Organizing, 
staffing^ and 
controlling 

-Training/De- 
velopment Audit 



There is an available state-^ 
ment of the objectives of 
training and development 
activity* This statement 
should be carefully written* 

Planning! prograiaming , bug- 
geting 

Policies should reflect and 
promote both enterprise and 
training and development 
activities* 

The policy manual should 
contain information relat" 
ing to the ^following: 
screening , selection , as- 
signment , training, eval- 
uation, and follow-up of 
all personnel* 

A procedures manual should 
contain detailed directions 
pertaining to the following t 
the selection, procurement, 
storage, and use of train- 
ing equipment and materials; 
orientation and upgrading of 
personnel, testing, and the 
identification of training 
techniques* The manual 
should bq .^xe^'iiCwe^ petlSSic- 
ally and * outmoded procedures 
should be eliminated* 



1* Satisfactory 
2* Needs Improve-- 
3* Unsatisfactory 



EVALUATION COMMENTS 
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ILLUSTRATION OF AN EVALUATION DESIGN FOR TRAINING AND DEVELOPMENT 
PROGRAMS (Principles and Criteria) 









BASIC COMPONENTS 


CRITERIA FOR MEASUREMENT 


RATING STANDARDS 


EVALUATION COMMENTS 


VII. Plant and Facili- 
ties 

-Existing plant 
facilities 

-Expansion of 
plant facilities 

-M aint en ance 


Determine basic planning 
requirements . 

Projections of training 
and plant facilities re- 
quirements should be 
based on an assessment of 
the condition and adequacy 
of existing buildings and 
other facilities. 

Plant and facilities plan- 
ning is coordinated with 
curriculuni planning and de- 
velopment; other types of 
services and operations. 

Plant and facilities re- 
.quirements and plans are 
coordinated with the over- 
all institutional or busi- 
ness enterprise require- 
ments aiid plans in all 
areas. 

Each institution should es- 
tablish with the assistance 
of competent consultants a 
master plan for the existing 
facility. 

Expansion should be consid- 
ered as long-range program- 
ming -so that plant develop- 
ment can proceed in an 


1. Satisfactory 

2. Needs Improve- 
ment 

3 . In need of im- 
med iat e att en - 
tion 

4. Unsatisfactory 
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ILLUSTRATION OF AN EVALUATION DESIGN FOR TRAINING AND DEVELOPMENT 
PRC^RAMS (Principles and Criteria) 











BASIC COMPONENTS 


CRITERIA FOR MEASUREMENT 


RATING STANDARDS 


EVALUATION COMMENTS 




orderly fashion , thus 
eliminating many problems 
before they occur. 

The development program 
should indicate space and 
function of all activities 
in terms of long-range 
planning. 

The duties and responsi- 
bilities for plant main- 
tenance should be clearly 
, defined and specifically 
assigned. In the interest 
of efficiency and economy 
the cost of maintenance 
and housekeeping should be 
constantly evaluated. 
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The preceding materials outlined some of the problems encountered in conducting 
effective evaluations. The exhibits included are designed to illustrate some criteria 
necessary in evaluating educational programs. 

The criteria used in making judgements about educational programs and social 
agencies* are representative of standard-oriented evaluations. Stake provides two 
bases for judging the characteristics of an educational program. The first standard 
concerns relative standards as reflected by characteristics of alternative programs; 
the second is with respect to standards as reflected by personal judgements. The 
evaluator^s cfompari^ons and judgements reflect relative comparisons and absolute 
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judgements, as illustrated in Figure 2. Rational judgement in educational evaluation 
is a decision as to how much to pay attention to the standards of each reference group 
in deciding' on a course of administrative action. This decision is more absolute in 
its comparison. Relative comparisons is accomplished in a similar manner but the stan- 
dards are not taken from reference groups. They are taken from descriptions of other 
programs. 

Figure 2 



A REPRESENTATION OF* THE PROCESS OF JUDGING THE MERIT 
OF AN EDUCATIONAL PROGRAM 



T 



DESCRIPTIVE DATAl 
from - - * - 

ONE P R06RAH 
t 

I 
1 



ABSOLUTE 
:OMPARISON 

4 ► 



STANDARDS 



RELATIVE 
COMPARISON 



III 



DESCRIPTIVE DAT/ 

from 

feNOTHER PROGRAM 




JUDGEMENTS 



♦source: -Robert E. Stake, "The Countenance of Educational Evaluation," 
Teachers College Record , Vol. 68, No. 7 (April, 1967), pp. 523-540. 
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The approach which we have briefly described illustrates one means of over*- 
coming evaluation problems in educational systems. The success of the approach is 
dependent upon evaluators recruited from outside the system rather than administrators 
of workers in the educational programs. In the next several pages, we will review and 
describe some approaches and models for evaluating varied types of training and de* 
velopment programs. We will review those showing greatest promise for reform in the 
traditional methods for assessing program effectiveness. 
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Section Three 



Program Evaluation Models 
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4.0 PROGRAM EVALUATION MODELS 



A number of approaches and models for evaluation have been developed. In 
seeking to conceptualize the various approaches to evaluation, rwo research models were 
described in a study by Schulberg and Baker. Characteristics and limitations of the 
goal--attainment model and the system model are described b^lov. 



MODEL 



CHARACTERISTICS 



LIMITATIONS 



Goal ^Attainment 



Integral components of this initial 
goal^setting, defining specific 
program objectives and goals in the 
evaluation process; selecting the 
appropriate methodology and criteria 
for determining the degree of suc- 
cess achieved in attaining the goal. 

The goal-attainment evaluation 
process not only proceeds to deter* 
mine measures of the goal, but col- 
lects data and appraises the effect 
of the goal. The initial goal is 
then modified on the basis of the 
collected data. 

The reseachor attempts to avoid bias 
of imposing his own objectives as 
criteria of the organization's ef*- 
fectivene&s by turning to the ad* 
ministrator of the program for a 
statement of the goals to be used 
as criteria. 



Tne assumption that 
specific goals can 
be evaluated and 
modified in isolation 
from other goals con* 
stitute an artificial, 
if not fallacious, 
approach. 

Much of the signifi- 
cance of the work is 
sacrificed in cases 
where program ad* 
ministrators do not 
intend to achieve the 
goals studied by the 
researcher. 
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System Model 
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The system model of evaluation is 
concerned with establishing a 
working model of a social unit 
which is capable of achieving a 
goal. Thi8 niodel is a multi- 
functional unit. 



Not readily adaptable 
to non*bureaucratic 
organizations or rela* 
tively loose organi* 
zational structures. 



f 
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MODBL 



CHARACTERISTIC? 



LIMITATIONS 



There is also recognition that an 
organization must fulfill at least 
£our functions: (l) the achieve* 
ment of goals and subgoals; (2) the 
effective coordination of organiza- 
tional subuhits? (3) the acquisition 
and maintenance of necessary re*- 
sources; and (4) the adaptation of 
thecrganxzation to the environment 
and to its internal demands. 

The system model establishes the 
degree to which an organization 
realizes its goals xinder a given 
set of conditions. Instead of 
simply identifying the goals of 
the organization and proceeding to 
study whether they are attained^ 
the system model requires a de<< 
termination by the analyst of 
what is considered to be a high- 
ly effective allocation of means. 

Consideration is qiven to feed- 
back mechanisms, i.e., the pro- 
cesses through which the effects 
of organisational actions are re- 
ported back to the organization 
and comp2u:ed vith desired per- 
formance . 



Time discrepancy often 
occurs between ad* 
ministrators and eval* 
uators. 

Problems of blocked 
feedback to the orga- 
nization. 



^e system model also suggests a 
variety of linkages and feedback 
mechanisms which can be used to 
bridge the gap between research 
findings and program modification. 
The model also has utility for 
determining the factors associated 
with effective integration of 
findings* 
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The preceding narrative portrayal of the goal^-attainment model and system 
model brings into focus particular characteristics of the models and describes some 
of the limitations inherent in the requirements for evaluation. 

Evaluation Is most effet^tive when it is a continuous process of program as'* 
sessment and improvement. It should be the kind of on-going process which focuses 
on the "why" of a program^ and supplies answers concerning why a program is succeeding 
or failing. A thorough appraisal of program activities requires an analysis of the 
attributes which contribute to success; the population exposed to the program^ in- 
cluding target subgroups affected; the environmental or situational context within 
which the program takes place^ locale^ etc.j and the kinds of effects produced by the 
program — cognitive^ attitudinal^ behavioral^ long or short-term^ unitary or multiple^ 
including any negative side effects. 

Evaluative research is at the core of good management ^ and constitutes an in- 
dispensable element in .the administrative process* In this vein ^ the administrative 
process is viewed as "cycle'* which includes such activities as^ 

decision-making 

programming 

communicating 

controlling 

re-appraising 

Accepting this cyclical concept of administrative activity^ then^ evaluation 
research becomes programmatic research whose major function is to aid administrators 
or program operators to plan and a'Jjust activities. Suchman identifies a sequence of 
development which relates to major aspects of programmatic activity*' He also states 
that there is constant interaction among them '•with success at one stage ^ and failure 
at any process forward to the next stage ^ and failure at any stage leading to a re- 
cycling to some earlier stage.** The sequence ^ in logical progression ^ follows: 

RESEARCH___^PLANNING DEMONSTRATION ^OPERATION 

According to Suchman^ this Process Model has been applied by Cuba and Clark to edu- 
cational programs. Research leaas to two stages of development (invention and design) 
which is followed by two stages of diffusion (dissemination and demonstration) which^ 
in turn^ leads to three stages of adoption (trial/ installation ^ and institutionaliza- 
tion) • He further advises that each step has its own objectives and appropriate 
criteria of success or failure* 
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There are also other approaches to evaluation and specialized models for par- 
ticular programs* Aronson and Sherwood discusses an evaluation model adopted by an 
organization called Opportunities for Youth (OfY) • The model adopted by this organi-* 
zation was that of an ^'action-research demonstration ^ the pxarpose of which was to 
discover effective and efficient ways of allocating resources in attacking social 
problems*"^ The action-*research demonstration contained the following features: 



AGENCY MODEL 

General Features of the Model 

a) An objective or set of objectives — the criteria by which the suc- 
cess or failure of the demonstration will ultimately be judged; 

b) A rationale — the li^e of reasoning by which it hopes to achieve 
its objectives (building upon selected variables believed to be cru- 
cial in dealing with the particular social problem^ the rationale 
explains Vhy changes in these variables will alleviate the problem; 

c) Strategies of avenues for intervention — programs suggested by the ^ 
rationale and the impact model that connects the program procedures 
and the kinds of changes the program is designed to produce; 

d) Evaluation -- the procedxires to determine whether the program achieves 
its objectives* 



The last features in the list above {Item d) calls for an evaluation of the 
effectiveness of the approach used to study behavior-change components* The research 
model for evaluation of action demonstrations with the above features has an experi- 
mental format* The chief characteristics of the experimental format relates to the 
control the experimenter has over salient variables. Thie experimental format permits 
designs such that^ ideally^ a researcher can limit the ntimber of variables^ and con- 
trol or manipulate them as well* An evaluator seeking to determine the cause and ef- 
fect relationships may design an experiment such that the variation in the independent 
or casual varlai^le mighty through experimental manipulation^ be maximized so as to 
render its impact* upon the idependent variable or effect more visible* 

One objective of experimental research is to secure two groups virtually 
identical in all things conceivably related to the dependent variable* In conducting 
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an experiment^ groups are exposed to programs/ situations / and special procedures / 
and note the effects. A test of knowledge is rendered at Time 1 {the first exposure 
to treatment or program activities) . After a given length of time, a second test of 
knowledge Is administered {Time 2) . The control group undergoes exactly the same kind 
of exposure as the experimental group except It is exposed to conventional activities 
rather than the resource type. A diagram- illustrating the comparison of the After 
versus the Before differences in an experimental format follows: 



Resources 



Before 



1 



After 



Experimental 


Knowledge 


Knowledge 


Group 


Time 1 


Time 2 


Control 


Knowledge 


Knowledge 


Group 


Time 1 


Time 2 



Conventional 



COMPAR£: 



Experimental (KTj - KTj^) 



vs. 



Control (kTj " KTj^) 



4.1 Basic Questions Regarding the Experimental Approach 

There are at least three questions raised regarding the experimental procedure: 



O 
CO 



Why do the groups have to be Identical? 
Ho^ are they made so? 
Why the control group? 

In reference to the first question, it Is asserted that if the groups are not identical 
in characteristics related to performance one does not know whether it was the treatment 
that causjed tlie effect or whether the effect Is attributable to the dlssimiliarity of the 
two groups. 

The second questlonof how to make the two groups identical is somewhat more dif- 
ficult to answer. A common strategy must be used to assign participants to the two 
groups. If the group has identical characteristics^ they can singly be assigned random- 
ly to each of the two groups. The assumption is that the two groups will be the same on 
all variables « Confidence Is placed in the technique of random allocation rather than 
pure simillarlty alone. Pair matching Is another means of equating groups, in which a 
replica for each person in the- experimental group Is found and placed In the control 
group* Vhere Is a limitation to this technique, however. It is very difficult to find 
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a limited ropulation who can be paired up in this way. The compromise strategy of fre- ' 
quency matching is often used as a substitute to pair matching. 

The third question, "Why the control group?*' can be answered by simply saying 
that the control group serves as a standard by which the resources or program activities 
acting as st^imuli — can be gauged. The standard activities or resources used in pro- 
grams are conventional situations or methods. Exposure to experimental activities by 
one group enables the program to be evaluated on its merits and potential for effective- 
ness . 

4.1.1. An Alternative Methodology 

Weiss and Rein discuss at length a role for experimental design in the evaluation 
of broad-aiia programs and sketch an outline of an approach considered to ba generally 
superior to experimental design as a methodology for evaluating programs.^'' The recom-* 
mended approach is characterized by the following; 

a) Process -oriented qualitative research 

b) Historical research 

c) Case study or comparative research ^ 

.CO 

The research approach is the same, but the first characterization (Item a) emphasizes 
the type of data which are collected; the second (Item b) emphasizes the method* s con- 
cern with the development of events through time; while the third emphasizes the utili- 
zation of a single case or small set of cases as a basis for generalization to a larger 
class. 

Several frameworks might prove useful for historical description, including 
those of system theory, of unfolding drama^'like events on which a plot structure can be 
imposed, and of an interaction of political forces. For program evaluation^ the ideas 
of system change or what is now being referred tq institutional change appear to be 
central, according to the authors, are designed to guide attention to the sort of events 
or happenings which should be observed and recorded in data-gathering, to the kind of in- 
quiries which must be answered in an analysis and interpretation of the data, and the 
kind of interfaces and/or connections which should be demonstrated in an evaluation 
report . 

A brief description of the systems approach and the dramaturgic approach follows: 

The Systems Approach - This approach has been discussed earlier in this report 
when a comparison was made between the Systems Model for evaluation and the Goal-Attain- 
ment Model. However, only characteristics and limitations of the models were compared. 
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with little detail of the basic, elements or components utilized in an evaluation. The 
systems framework is generally useful in the conceptualization of events and other 
phenomena considered pertinent to an inquiry^ in determining the scope of an investiga- 
tion ^ in suggesting the roles to be played by various actors ^ and in providing general 
ideas relative to interaction between the actors. 

The systems approach involves; 1) delimiting the scope of the inquiry in terms 
of time and energy? 2) identifying systems in relation to issues ^ and different issues 
might well require different system definitions. For example^ the system which must be 
studied in atten5)ts to evaluate a Model Cities planning program would be different from 
the system which would be studied in connection with a police department Community Re- 
lations program. 

The Dematurgic Approach - is most useful in describing small-scale events ^ and 
for relating Individual actions to situational outcomes* Politically^ the approach is 
used to describe a series of events which occur over time^ involving a substantial nvonber 
of persona. The dramaturgic framework involves a framework similar to what has been re- 
ferred to as "methodological individualism" which seeks to explain events by reference 
to the action of individuals within situations* It begins with the construction of a 
story line involving actors within setting^ often engaging in coalitions and conflicts ^ 
and then proceeds to recording interactions which form plots and subplots which move to 
some resolution. Historians have used this type of evaluative technique in reference 
to social events f ideological conflicts ^ economic and political movements. 

4.1*2. The Political Process 

The political process involves a framework which takes into account a connected 
series of events as well as observations of single events* It is considered to be more 
complex than the dramaturgic approach. Actors m this perspective are perceived to be 
interest groups who express or subscribe to some kind of strategy. An issue in evalua- 
tion of such programs is to determine how groups mobilize resources in response to 
gram intervention^ what ways they commit themselves to affecting events^ and the extent 
of success of commitments made. * 

In analyzing data for each approach^ three levels of generalization may be used. 
The first level of generalization is one involving the organization of materials to pro- 
vide for the description of what happened in concrete cases. The second level of ab- 
straction would describe the types of systems^ structures ^ or processes involved* The 
third level is one which would require the presentation of a general model for under- 
standing of the findings and what consequences are involved in introducing a program 
for change* 
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4*2 Evaluating Social Programs , 

According to Greenberg^ "governmental programs which provide social and educa- 
tional searsrices to the public are generally costly in terms of money and manpower* "12 
The concept of accountability as revealed through accomplishments versus funds spent 
looms large in the whole arena f * * social programming* An evaluation of social programs 
is defined as measurement of accc ^lishment with-respect to a program's particular tar- 
getr meaningful si:udy of goals^ o<^jectives, program activities^ and overall program 
operations as they attempt to satisfy the program goals and objectives* 

One model for social prbgrsumning which has been utilized in evaluating action 
or community action programs can be labeled "a Social Action Program Management informa- 
tion System (SAP-MiS).** Fundamentally^ the SAP management information system is a method 
of providing meaningful information to managers ^.t all levels in carrying out an inte- 
grated program operation* Thic Model is more applicable to such programs as the Communi- 
ty Action Programs r Model Cities Programs, and more individualized program components 
such as Manpower Training, Economic Development, and Health Care* Procedural aspects of 
the model may be briefly illustrated by the following checklist; 

✓Defining the needs of poor people 

✓Setting objectives for meeting these needs ^ 

CQ 

fc^Analyzing and selecting alternative courses of action for achieving the 
objectives 

✓Allocating resources among alternatives 

✓Assigning individual responsibilities for carrying out the approved courses 
of action* 

✓Monitoring progress against plans ^ both to improve individual performance 
and to strengthen plans in each recycling of the planning process 

These are actual steps in the overall process of the SAP Management Information System* 
Progress reports are prepared during the monitoring of programs and, as such^ provide ^ 
feedt>ack on progress in carrying out approved plans and in achieving approved objectives* 
The monitoring of program performance and the reporting of what progress is made may in- 
volve five major features: 1) Incorporation of other related programs; 2) standardized 
information categories; 3) consistency at all organizational levels; 4) permanent struc^ 
ture Within which refinements and improvements can be made; and 5) an empirical approach 
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to measuring results* The latter feature represents the evaluation phase of the Model* 

The empirical approach to measuring program results should initially incorporate 
basic information categories* Data to be collected would satisfy requirements for the 
following questions t 

^ Questions to Measure Program Results 

1) \Ihat services are you offering, and what other Federal, state, and local 
programs are you coordinating in meeting the needs of poor people? 

2) How many poor people participated in each program and what are their 
basic characteristics? 

' 3) >ftiat happened to the participants when they completed the programs? 

4) What did it cost to provide the services? 

5) Were the goals of the program achieved? 

Where possible the progress reports should include measures of program results^ 
These measures are ^'empirical*' in the sense that the SAP*MIS is based on an empirical 
appjroach in that it attempts to monitor the progress of programs through a series of 
specific steps designed (in the overall planning mechanism) to alleviate conditions of 
deprivation* Indicators of program impact have to be developed and built into the Social 
Action Program management information systems as they are proven to be relxable measure* 
ments of progress in meeting the needs of people* As an illustration, a high school 
dropout who is c ontacted through a neighborhood center outreach function may be inter- 
viewed by an intake counselor, placed in a prevocational training program, and then 
placed in a job by a social agency or the state employment service* The HIS is designed 
to monitor each step taken as the individual escalates toward economic^ security and in-* 
dividual improvement* In social action programs, this kind of monitoring is one way of 
demonstrating tangible results of an experimental training and development program* 

Sevei^al exhibits (I, II, III) have been included as examples of forms used for 
program monitoring* These forms are used to demonstrate procedures that a Community 
Action Agency might use in c ollecting key items of information on participants, record- 
ing services provided each participant, and preparing progress reports oh Social Action 
Program (SAP) management use* Figure 3 illustrates steps involved in a typical Coimnuni- 
ty or Social Action Agency Participation Information System* 
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The general process steps a Community Action Agency or other social agencies 
might establish to provide progress information are: 

Step 1. Survey of candidate needs . This step involves obtaining basic infor- 
mation on individual poor people with which to assess needs, review 
progress over time, and summarise participant information for manage* 
ment use. 

Step 2. Ir formation Processing . Participant information is used in at least 

tEree ways: T) to assist planners in defining the needs of target groups 
and in planning programs to meet these needs; 2} to provide counselors 
with useful background information prior to interviews with potential 
participants; and 3) to provide information for assessing and reporting 
program progress and participant characteristics. 

Step 3. Program Planning Process . A summary report of candidate needs is pre- 
pared'. This and related information is made available to planners so 
that they can identify the overall needs of poor people and to plan 
programs to meet these needs. 

step 4# Intake Counseling . Reviewing the background and status of current or ^ 
potential participants • 

step 5. Program Administration * The program administrator or his staff assis- 
tants will receive referral slips, and \*ill maintain basic records on 
participants and all actions taken. Participant information will be 
used in preparing a monthly program participation review. 

step 6. Report Preparation . The program process report is a summary of data 
contained In program participation reviews. 

step 7. Management . The Program Director or tnanagement analyse and evaluate the 
program participa^.:ion review. This permits the assessment of all de~ 
cisions affecting re-planning and re -programming actions. 



step 8# Final Review of Program Progress . The program progress reports will be 
reviewed in detail oy regional and headqtiarters managers, and made de- 
cisions relative to opportunities to provide meaningful assistance, and 
assess re^planning and refunding implications* 
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To monitor progrsun progress of social acti^on prpgrams, several ways are available 
for developing plans for a management information system* One way of developing an ef- 
fective system for management use Involves the following: 1) relate existing components 
to the standardised categories of service; 2) identify the information required for re- 
porting to the social action program component for each category of service; 3) define 
other information required for agency management use; 4) identify sources of required 
information; 5) gain agreement on the appropriate information requirements with each 
agency or unit; 6) develop foirmal procedures for obtaining information from the sources 
identified in step 4 above; 7) guide sources in preparation of the information; and 8) 
begin reporting specific information required by the program management. 

As a conclusion to this proposed model for evaluating social action and related 
programs, it is necessary to identify all of the items of information needed for evalua- 
ting program effectiveness. Evaluation of a social program is more closely related to 
and identified as a research function rather than as a service function. As such the 
final index for evaluation is* 

Efficiency = Output (in terms of goal fulfillment ) 
Input tin terms of dollars/ services 



servxces and/or personnel time) 
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Section Four 



Other Approaches to Evaluation 
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5.0 OTHER APPROACHES TO EVALUATION 



There are other approaches being utilized in evaluating programs ^ both formal 
and informal. Informal evaluation depends on casual observation ^ implicit goals ^ in- 
tuitive normsr and subjective judgement. Formal approaches^ similar to the preceding 
Model fbr Social Action Programs ^ emphasize inputs and outputs. 

^ In general^ major .erqphf^sis of most evaluative studies will be upon the demon- 
stratio^i and operational stages of programs. Research and planning may also be evalua 
ted, perhaps, to a lesser degree unless the major function of the organization is re- 
search QT planning. 



5.1. Evaluative Criteri.a 

It is necessary for evaluators to find reliable and valid operational indices 
for measuring the attainment of objectives. Evaluative criteria represent the basis for 
decision**making. In brief, the decision-making process is div^ided into three components, 
each of which requires different criteria of effectiveness:^^ ^ 

5.1.1. An Information Component - A reference system, model, process, or stan*- 
dard according to which information may be collected and evaluated. It 
requires action rather than construct relevance. 

5.1.2. An Interpretation Component - A value position or preference system repre- 
senting the desirable, the appropriate, the acceptable means and ends for 
making decisions. This requires a weighting process by which alterna- 
tives are assigned priorities. 

5.1.3. An Action Component - A criteria function designed to set the limits of 
acceptable error. This represents an operational process or administra*- 
tive rule by which error costs are matched against potential outcomes. 

The first component entails information retrieval and utilization systems? the 
second, decision-mahing systems;and the third, risk-taking choice situations. Each 
element or component, though opetational in character, reflects a relationship between 
evaluative research and program management processes. 

According to Sunhman, evaluative criteria may be used for the following purposes: 
X) To monitor a steady st.ate so as to determine when a correction is needed (as in an 
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automated system involving servomechanisms) ; 2) To identify alternatives in a problem 
situation; 3) To weigh alternative courses of decision-making in terms of relative gains 
and losses; 4) To determine corrective action and error-risks involved in various ap- 
proaches to change:. 

5. 2 External and Internal Criteria 

Both external and internal criteria may be applied in evaluation studies. Ex - 
ternal criteria are used to measure the results of programs. By applying external cri- 
teria" " reports^ observations^ interviews^ questionnaires^ work samples^ and statistics -- 
the evaluator can determine the value of a program to the organization. Value ^ as used 
here, is stated in terms of orgaiiizational benefits and can be translated into dollars 
or numerical indexes of gain or loss. The cost*benefit analysis used by economists is 
usually a useful approach to determining cost estimates. 

Internal evaluation may involve participation measures ^ norm comparisons, measures 
of behavioral change, participant reactions ^ experiiviental research^ and measurement against 
specific standards. 

5. 3 Forms of Evaluation ^ 

15 

Evaluation studies may take on several forms: 1) effect studies; 2) opera- 
tions analysis; 3) surveys o f need; and 4) investigations ^ or a combination of one or 
more of the forms. Riecken explains these principal forms of evaluation. In an in- 
ternal memorandum to the Ford Foundation ^ Riecken gives a rough typology in which method 
and purpose are combined, and suggest that evaluation studies take the following princi- 
pal forms: 

Measuring Effects — Studies of effects represent the maximum contribution 
made by social scientists to evaluative methodology. This form of evaluation 
involves finding or devising techniques for detecting change in subjects and 
estimating the degree to which observed phenomena approach the objectives of 
the program. Studies of effects are designed to feed back results in program 
planning or policy-making. The focus of attention is on e nds t 

Various research designs for studying the effects of programs in terms of the 
** cause/effect^ process have been proposed. Basically, the designs presented reflect 
some features of the- experimental or systems model. Suchman outlines what he considers 
to be the logical requirements of an evaluation '^tudy. Using the familiar model of 
the laboratory experiment, he diagrams the "before" and "after" measures this way: 



5.3.1. 
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Before After 



EXPERIMENTAL 


^1 


X2 


d = Xj - 




CONTROL 


X^ 


X^ 
2 


d^ = xj - X^ 





It should be noted that the experimental approach has been discussed in greater details 
in the section describing the *'Agency Model.'' 

5.3.2. Operational .Analysis — This form of evaluation places emphasis on means or 
operations of the program without specific attention to ends. This type of 
evaluation has two forms: 1) conspliance with standards; and 2) the periodic 
reporting of activities without reference to pre -determined standards. It is 

a kind of administrative evaluation in which reports of activities are used in 
connection vith program jiistification. For instance^ budget sections base 
their evaluation of requests for funds partly on activity reports. 

■4 

5.3.3. Surveys of Need — - assessing the need for or desirability of a contemplated 
action or potential course of action. 

5.3.4. Investigations — This is a form of evaluation that is often referred to as an 
"independent audit, and is most often an attempt to determine the current status 
of operations and achievements of a program. 



The preceding materials have covered models applicable to general training and 
development programs, educational programs, agency and other institutional programs, 
and social action projects. They are by no means comtprehensive in the treatment of 
evaluative methodologies. The intent of all exhibits, illustrations f and other data is 
to provide direction to effurts by those concerned with improving means of assessing 
social programs more effectively. 



The final portion of this handbook is devoted to a summary of steps for identify^ 
ing and measuring t.he changes and results of traininq and development projects. 
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IMPORTANT STEPS IN INDICATING THE 
RESULTS OF AN EVALUATION 
(A 'Check List' of Principal Actions) •'•^ 



Step 1. DESCRIBE THE DEVELOPMENT PROJECT AND SPECIFY ITS GOALS IN A WRITTEN STATEMENT , 
COVERING THE FOLLOWING: 

1. Nature of the project and specific operations to be carried, include an 
. organizational chart to show the administrative and staff positions. 

2* Kinds of results sought — including impersonal, organizational, personal 
(information, attitudes, skills, actions, habits, social relationships, 
etc*} changes in conditions, capital resources, and program guidelines* 

3* Area of activity and area of influence* 

4* Program objectives, activities and goals* 

5* Criteria or standards for measxirement* Indicate evaluative model* 

6* Intervening varitUDles or side effects anticipated* 

7* Cost analysis, based on budget information and program activities* 

This written statement or evaluation design should be reviewed with the funding 
agency, interested parties, to the extent practicable* 

Step 2* DECIDE WHAT KINDS OF DATA WILL BE USED TO INDICATE PROJECT RESULTS AND HOW 
THESE DATA ARE TO BE OBTAINED: 

1* Find out what registration and census data are already available or al- 
rea<^ being collect:ed, and decide^ whether it can be utilized and how such 
data can be obtain^.d* 

2* Decide what individuals or organizations are to be asked to collect ad** 

ditional pertinent data, including^ persons directly involved in the project 
or program (built-in data collection); and what particular reports are to 
be requested* 
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3* Decide what data are to be collected by systematic questioning of af- 
fected (program participants ^ clients) or control groups ^ and plan the 
questions to be asked* Prepare a standardized interview schedule or 
quest ionnaire^ if one is tp be used* Items included on the form should 
reflect data needed 'to effectively determine program effectiveness* 

Step 3* COLLECT THE DESIRED DATA 

1* Beforehand — to help plan the project* 
2* During — to help administer the project* 
3* After — to help plan future project* 

Step 4* ANALYZE AND INTERPRET THE FINDINGS, AND REVIEW FINDINGS AND INTERPRETATIONS 
WITH THE VARIOUS INTERESTED PARTIES* 



Some sample forms for particular types of evaluation have been included in the 
appendices* Also included is a rather comprehensive bibliography which may be quite 
useful to a person conducting various types of evaluations* 
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Appendix A 
FOOTNOTES 



Francis G* Caro; Readings in Evaluation Research * 
2 

William R* Tracey^ Evaluating Training and Develoianent Systems * American 
Management Association^ lnc*7 1968^ pp* 13*'14* 

* 3 

* . * Edward A* Suchman^ "Evaluating Educational Programs The urban Review ^ Center 
for*Urban Education^ Vol* 3^ No* 4 (February^ 1969) ^ pp* 15-17, 44* 

^Francis G* Caro^ Readings in Evaluation Research * 

m 

William R* Tracey^ opcit ^ p* 16* 

* 

^Herbert c* Schulberg and Frank Baker^ "Program Evaluation Models and the Iraple- 
mentation of Research Findings^" American Journal o£ Public Health * Vol* 58^ No* 7> 
July, 1968, pp* 1248-1255* 

7 00 
Edward A* Suchman, opcit * ^ 

Sidney H* Aronson and Clarence C* Sherwood, "Researcher versus Practitioner: 
Problems in social Research,** social Work , Vol* 12, No* 4 (1967), pp* 89-96* 

a 

^Dennis P* Forcese and Stephen Richer, social Research Methods * New Jersey: 
Prentice*Hallr 1973. PP* 88*93* 

^^Robert S* Weiss and Martin Rein, **The Evaluation of Broad Aim Programs: »if* 
ficulties in Experimental Design and an Alternative,** in Evaluating Action Programs , 
Boston: Allyn and Parson, 1972, pp* 236-249* 
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^^J.W.N. Watkins, "Ideal Types and Historical Explanation," in Herbert Feigl 
and Mary Brodbeck (eds.K Readings in The Philosophy^ of Science . New York* Appleton- 
Century*Crafts , 1971. 

12 

B.G. Greenberg, "^Evaluation 'of Social Programs** in Caro, op. cit . , pp* 155-174* 

^•'por full explanation of this procedure, see: Manage m ent In fo rmation Reporting 
by Community Action Agency , U.S. Office of Economic Opportunity , 1966 . 

^'-^Kdward A. Suchjn€Ui, ** Action for What? A Critique for Evaluative* Research," 
in The Organization, Management, and Tactics of Social Research , edited by R. 0*Toole. 
Cambridge, Massachusetts: Schenkman Publishing Co., 1970, in Carol H. Weiss, Evaluating 
Action Progreuns , pp . 72*73 . 

^^Henry W. Riecken, "Memorandum On Program Evaluation," in Carol H. Weiss, 
Evaluating Action Programs . Boston: Allyn and. Bacon, inc., 1972, pp. 85-87. 

^^See: Samuel P. Hayes, "Evaluating Development Projects," Published by UNESCO, 
1959, pp. 92*93. 



Appendix B 



BIBLIOGRAPHY 

ABRAMS. MARK A. 

1951 Social Surveys and social Action . London^ Heinemann. 

ALKIN, MARVIN 

1969 "Evaluation Theory Development." Evaluation Ccmment ^ vol. 2, no. 1^ pp. 2-7, 

AMERICAN INSTITUTES FOR RESEARCH* 

1970 Evaluative Research Strategies and Methods ^ Pittsburgh: JLmerican 
Institutes for Research. 

AMERICAN PUBLIC HEALTH ASSOCIATION* 

1955 Health Program Evaluation^ Public Health Reports ^ vol. 70^ no. 2. 

ANDREW. GWEN* 

1967 "some Observations on Management Problems in Applied social Research." 

The American Sociologist , vol. 2. no* 2. pp* 84-89. 92. J? 

ARNOLD. MARy F* 

1963 Principles of Evaluation, in: The Voluntary Health Agenc_y ^ Getting Commu- 
nity Action ^ .p* 45*9* (Continuing Education Monographs No. 4) San Francisco^ 
Calif*. Western Regional Offices. APHA. 

ARONSON, SIDNEY H* . and CLARENCE C* SHERWOOD* 

1967 "Research Versus Practitioner: Problems in Social Action Research*" Social 
Work. vol. 39^ no* 4. pp. 89-96. 

BAKER. ROBERT L. 

1969 "Curriculvmi Evaluation." Review of Educational Research , vol. 39. no. 3. 
pp* 339*358. 



39 



BATTEN, 

1956 Some Criteria for the Evaluation of the Training of Extension Workers in 

Fundamental Education Programmes* International Journal of ftdult Education * 
vol* 17 , no* 3, September* pp* 12-19* 

BARTON, ALLEN H* 

1959 Studying the Effects of a College Education * New Haven; The Edward H* 
Hazeu Foundation* 

BATEMAN, WORTH* 

1368 ''Assessing Program Effectiveness: A Rating System for Identifying Relative 
Program Success*" Welfare in Review ^ vol* 6, no* 1, pp* 1"10* 

BELSKAW, CYRIL S* 

1966 "Evaluation of Technical Assistance as a Contribution to Development*'* 

International Development Review ^ vol* 8, pp* 2-23* 

BENEDICT, BARBARA A*, PAULA H* CALDER, DANIEL M* CALLAHAN, HARVEY HORNSTEIN, and 
MATTHEW B* MILES* 

1967 "The Clinical-Experimental Approach to Assessing Organizational Change 

Efforts* *' Journal of Applied Behavioral Science # vcl* 3, no* 3, pp* 347-380* 

BENNIS/ WARREN* 

1965 '*Theorjf and Method in Applying Behavioral Science to Planned Organizational 

Change* " Journal of Applied Behavioral Science ^ vox* 1, no* 4, pp* 337-360* 

BERLAK, HAROLD* 

1970 "Values, Goals, Public Policy and Educational Evaluation*** Review of Educa- 
tional Research ^ vol* 40, no* 2, pp* 261-278* 

BIGMAN, STANLEY K* 

1961 "Evaluating the Effectiveness of Religious Programs*" Review of Religious 
Research , vol * 2 , no * 3 , pp * 97- 12 1 * 



41 



BLEHKNERr MARGARET* 

1950 "Obstacles to Evaluative Research in Casework." Parts 1 and 2. Social 
Casework f vol* 31/ nos* 2-3r pp* 54-60/ 97-105. 

BLUM/ HENDRIK L* / and ALVIN R. LEONARD 

1963 "Evaluation Research and Demonstration." in Public Administratio^it A Public 
Health Viewpoint * New York: The Macmillan Company^ 1963r pp* 286-322* 

BOGARDUS/. EMORY S* 

1936 Introduction to Social Research ^ Los Angeles/ Calif*/ New York/ Suttonhouse/ 
1936* 

BORGATTA/ EDGAR F* 

1966 "Research Problems in Evaluation of Health service Demonstrations*" Millbank 
Memorial Fund _Quarterly f vol* 44, no* 4/ part 2/ pp* 182-199* 

BORUS, MICHAEL E* / and WILLIAM R* TASH ^ 
1970 Measuring the Impact of Manpov;6r Prograu^s : A Primer * Policy Papers in ^ 
Human Resources and industrial Relations* No* 17; Ann Arborr Institute 
of Labor and industrial Relations* 

BRIM/ ORVILLE G*/ JR* 

1957 "Evaluating the Effects cf P?%on1. Education*" Journal of Marriage and 
Family Livi n g / vol* 19/ pp* 54^-60* 

BROOKS/ MICHAEL P* 

1965 "The Community Action Program as a Setting for Applied Research*" Journal 
of Social Issues/ vol* 21/ no* 1/ pp* 29^40* 



BRUNNERr EGMUND DES 

1967 "Evaluation Research in Adult Education*" International Review of Comrounity 
Development , nos* 17-18/ pp* 97-102* 



42 



BUCKLE7r WALTER 

1967 Sociology and Modern Systems Theory * Englewood Cliffs, New Jersey: 
Prentice-Hallr Inc* 

BURGESS r A* 

1961 Nutrition Education in Public Health Programs - what Have We Learned? 
Amer^jan Journal of Public Health y no* 51^ p* 1715* 

BURROUGH, G* E* R* 

1956 Evaluation in Fundamental Education^ international Journal of Adult 
Education ^ vol* 17^ no* 4^ PP* 6-13* 

BYNDERr HBRBERS! 

1966 "Sociology in a Hospital: A Case Study in Frustration*" In Arthur B* 

Shostak (ed*) Sociplpgy in Action * Homewoodr Illinois: Dorsey Press^ 
PP* 61-70* CO 

iO 

BYRN, DRRCIE (ed*) 

1959 Evaluation In Extension * Washington^ D*C*r United States Department 
of Agriculture^ Division of Extension Research and Training^ Federal 
Bxtension Service^ 

CALDWELL, MICHABL S* 

1968 **An Approach to the Assessment of Educational Planning*" Educational 

Techno logy f vol* 8, no* 19* p* 5-12* 

CAMPBELL, DONALD T* 

1967 "^Administrative Experimentation, institutional Records* and Nonreactive 

Measures*'* In J*C* Stanley (ed*) Improving Experimental Design and 
Statistical Analysis * Chicago: Rand McNally £t Co** pp* 257-291* 




43 



CAMPBEIJ>, DONALD T. 

1957 "Factors Belevant to the Validity of Experiments in Social Settings." 
Psychological Bulletin , 54: 297-312. 

* 

CAMPBELL, DONALD T. 

1963 "Prom Description to Experimentation interpreting Trends in Quasi- Experiment. " 
in Chester W. Harris (ed.), Problems in Measuring Change . Madison: 
University of Wisconsin Press. 

CAROi FRANCIS G. 

1969 "Approaches to Evaluative Research^ A Review." Human Organization j 
vol. 28# no. 2, pp. 87-99. 

CHAPIN, P. STUART 

1957 Experimental Designs in Sociological Research . New York: Harper and 

Brothers (revised 1955) . ^ 

CEffiRNEY, PAUL R. (ed.) 

1971 Making Evaluatioti Research Useful . Columbia* Maryland! American 
City Corporation. 

CHERNS, A. 

1969 "The Use of the Social Sciences." Human Relations * vol. 21, no. 4, pp. 313-325. 
COHEN, DAVID K. 

1970 "Politics and Research^ Evaluation of Social Action Programs in Education." 

Review of Educational Research * vol^ 40, no. 2, pp. 213-238. 

COLVIN, C. R. 

1968 "Reading Program that Failed^ or Did it? Hawthorne Effect." Journal 
of Reading * vol. 12* no. 4, pp. 142-146. 



i 



COMMUNITY COUNCIL OF GREATER NEW YORK, RESEARCH DEPARTMENT 

1967 Issues in Community Action Research. Report of the Spring Research Forum 
on Evaluation Efforts in Three New York City Community Action Programs. 
Ntew York I The Council* 

CRONBACH, LEE 

1968 "Evaluation For Course Improvement.*' Teachers College Record ^ vol. 64, 

no* 8, pp. 672-683* 

DAILY, EDWIN F*, and MILDRED A* MOREHEAD 

1956 "A Method of Evaluating and Improving the Quality of Medical Care*" 
American Journal of Public Health / vo 1 . 46 # no . 7 , pp . 848-854 . 

DAVIS/ JAMES A* 

1964 "Great Books and Small Groups: An informal History of a National Survey.'' 
In Phillip E. Hammond (ed*) Sociologists at Work; Essays on the Craft 
of Social Research i New York: Basic Books* Inc., Publishers, pp* 212*234. 

DENISTON, 0*L*,.I*M* ROSENSTOCK, and V.A. GETTING 

1968 ''Evaluation of Program Effectiveness.'' Public Health Reports^ vol. 83/ no. 
pp* 323-335* 

DONABEDIAN/ AVEDIS 

1966 "Evaluating the Quality of Medical Care*" Millbank Memorial Fund Quarterly , 
vol. 44, no. 3/ part 2, pp. 166-203* 

DEXTER, LEWIS A. 

1966 "Impressions about Utility and Wastefulness in Applied Social Science 
Studies* American Behavioral Scientist , vol. 9, no. 6/ pp* 9-XO* 



45 



DORFMAN, ROBERT 

1965 "Introduction." In Measuring Benefits of Government Investments . 
Washington^ D.C.: The Brookings Institution^ pp. 4-22. 

DOWNS, ANTHONY 

1965 "Some Thoughts on Giving People Economic Advice." American Behavioral 

Scientist f vol. 9, no. 1, pp. 30-32. 

DRESS EL, PAUL L. (ed.) ^- 

1961 Evaluation in Higher Education . Boston: Houghton Miff lin* Company. 

DREW, ELIZABETH B. 

1967 "HEW Grapples with PPBS.** The Public Interest , no. 8, pp. 9-29. 

DUBOIS, PHILIP H. , and E. DOUGLAS MAYER (eds.) 

1971 Research Strategies for Evaluating Training . AERA Monograph Series on 
Curriculum Evaluation, no. 4, Chicago: Rand McNally & Co. 

DYER, H. S. 

1966 "Pennsylvania Plant Evaluating the Quality of Educational Programs.*' 

Science Education , vol. 50, no. 3f pp. 242-248. 

EATON, JOSEPH W. 

1962 "Symbolic and Substantive Evaluation Research." Administrative Science 

Quarterly , vol. 6, no. 4, pp. 421-442. 

EiXJCATIONAL TESTING SERVICE 

1966 On Evaluating Title I Programs . Princeton: Educational Testing Service. 

EIDEUi, TERRY L. , and JOANNE M. KITCHEL 

1968 Knowledge Production and Utilization in Educational Administration . 
Eugene: Center for the Advanced Study of Educational Administration, 
University of Oregon. 



ERLC 



46 



ELINSON, JACK 

1967 "Effectiveness of Social Action Programs in Health and Welfare. ** In 
Assessing the Effectiveness of Child Health Services (Report of the 
Fifty-sixth Ross Conference on Pediatric Research). Columbus, Ohio: 
Ross Laboratories r pp* 77-88* 

ETZIONI, AMITAI 

1960 "Two Approaches to Organizational Analysis; A Critique and a Suggestion." 
Administrative Science QuarterlV f vol* 5r no* 2 r pp* 257-278* 

EVANS, JOBN W* 

1969 ''Evaluating Social Action Programs*" Social Science Quarterly ^ vol* 50, 
no* 3, pp* 568-581* 

FAIRMEATHER, GEORGE 

1967 Methods for Experimental Social Innovation * New York: John Wiley & Sons^ £>• 
Inc* r pp* 24*36* ^ W5 

FELIiIN, PHILLIP, TONY TRIPO0I, and HENRY J* MEYER (eds.) 

1969 Exemplars of Social Research * Itascar Illinois* F.E* Peacock Publications* 

FERMAN, IX)UIS A, 

1969 "Some Perspectives on Evaluating Social Welfare Programs*" Annals of the 
American Academy of Political and Social Science ^ vol. 385r pp. 143-156. 

FIRST NATIONAL CONFERENCE ON EVALUATION IN PUBLIC HEALTH 

1960 Education Series No* 89* Ann Arbor: University of Michigan, School of 
Public Health. 



FLANAGAN, JOHN C. 

1966 "Evaluating Educational Outcomes*" Science Education , vol* 50r no* 3, 
pp* 248-251* 



47 



FLECK, ANDREW C , JR. 

1961 "Evaluation as a Logical Process." Canadian Journal of Public Health , 
vol. 52, no. 5# pp. 185-191. 

FOKi DAVID J. 

1967 "Issues in Evaluating Programs for Disadvantaged Children." Urban Review # 
vol. 2, pp. 7# 9, ll. 

FREEMAN, HOWARD E. , and CLARENCE C. SHERWOOD 

1965 "Research in Large-Scale Intervention Programs." Journal of Social Issues * 
vol. 21, no. 1# pp. 11-28. 

GETTING, VLADO 

1957 "Part II-Evaluation. " American Journal of Public Health , i^ol. 47# no. 4, ^ 
pp. 409-413. ^ 

GLASER, EDWARD M. and HUBERT S. COFFEY 

Utilization of Applicable Research and Demonstration Results . Los Angeles: 
Human Interaction Research Institute, n.d. 

GLASS, GENE V. ^ 

The Growth of Evaluation Methodology . AERA Monograph Series on Curriculum 
Evaluation* no. 7. Chicago: Rand McNally & Co., in press. 

GLENNAN, THOMAS K. , JR. 

1969 Evaluating Federal Manpower Programs : Notes and Observations . Santa Monica: 
The RAND Corporation. 

GLOCK, CHARLES Y. et. al. 

196 L Case Studies in Bringing Behavioral Science Into Use: Studies in the 
Utilization of Behavioral Science . Vol. 1. Stanford: Institute for 
Communication Research. 



48 



GOLLIN, ALBERT £. 

1970 "The Evaluation of Overseas Programs: Applied Research and Its Organisational 
Context." in Robert Campbell, Bert King, and John Nagay (eds*) Education 
and Training for International Living : Concepts . Arlington, Virginia, 
Beatty Publishers* 

GORHAM, WILLIAM 

1967 "Notes of a I'ractitioner . " The Public Interest , no. 8, pp. 4-8. 

GREeNBERG, BEBmRD Gw and BEFWYN F. MATTISON 

1955 "The Whys and Wherefores of Program Evaluation." Canadian Journal of Public 
Health ,, vol. 46, no. 7, pp. 293-299. 

GRIESSMAN, B. EUGENE 

1969 '*An Approach to Evaluating Comprehensive Social Projects." Educational 
Technolog y, vol. 9, no. 2, pp. 16-19. 

GROBMAN, HULDA 

1968 Evaluation Activities of Curriculum Projects ^ A Starting point . AERA 
Monograph Series on Curriculum Evaluation, no. 2, Chicago: Rand McNally St Co. 

GRUENBERG, ERNEST M. (ed.) 

1966 "Evaluating the Effectiveness of Mental Health Services." Millbank Memoria l 
Fund Quarterly , vol. 44, no. 1, part 2, whole issue. 

GUBA, EGON G. 

1968 ^'Development, Diffusion and Hfvaluation. " In Terry L. Eidall and Joanne M. 

Kitchel (eds . ) Knowledge Production and Utilisation in Educational Adminis* 

t rat ion . Eugene* Center for the Advanced Study of Educational Administration, 

University of Oregon, pp# 37-63 



49 



HAGEN, ELIZABETH Pw and ROBERT THORNDIKE 

I960 "Evaluation*" Encyclopedia of Educational Research * 3rd ed* New York; 
The Macmillan Company, pp* 483*486* 

HALL, RICHARD H* 

1966 "The Applied Sociologist and Organizational Sociology*" in Arthur B. 

Shostak (ed*) Sociology in Action * Homewood, Illinois; Dorsey Press, 
PP* 33-38* 

HARDIN, EINAR, and MICHAEL E* BORUS 

1966 "An Economic Evaluation of the Re-Training Program in Michigant Methodological 
Problems of Research*" Proceedings of the Social Statistics Section * 
Washington, D.C*; American Statistical Association, pp* 133-137* 

HASTINGS, J. THOMAS 

1968 "Curriculum Evaluation; The Why of the outcomes*" Journal of Educational 

Measurement s vol* 3, no* 3/ pp* 27*32* Also reprinted in Normal Gronlund (ed*) 
Readings in Measurement and Evaluation * New York; The Macmillan Company, 
pp* 53^60* 

HAVELOCK, RONALD C* 

1969 Planning for Innovation through Dissemination and Utilization of Knowledge * ^ 
Ann Arbort Institute for Social Research, University of Michigan* ^ 

HAYES, SAMUEL 

1959 Measuring the Results of Development Projects* Paris UNESCO* 



HEMPHILL, JOHN K* 

1969 "The Relationships between Research and Evaluation Studies*" In Ralph W* 

Tyler (ed*) Educational Evaluation! New Roles, New Means * 68th Yearbook 
of the National Society for the Study of Education* Chicago* The Society, 
1969, pp* 189^220* 



50 



HERMAN, MELVIN 

1965 "Problems of Evaluation*" The American Child / vol* 47 (March) pp* 5-10* 

/ 

HERZOG, ELIZABETH 

1959 Some Guidelines for Evaluative Research * Washington, D.C* : Government 
Printing office* 

HESSIELING, P« 

1964 "Principles of Evaluation*" Social Compass , vol* 11/ no* 1, pp* 5-22* 

HILL, MARJORIE J*, and HOWARD T* BLANE 

1967 ''Evaluation of Psycho*therapy with Alcoholics*" Quarterly Journal of 

Studies on Alcohol , vol* 28, no* 1, pp* 76-104* 

HOUilDAY, L. P. ^ 
1969 Appraising Selected Manpower Training: Programs in the Los Angeles Area * 
Santa Monica t The RAND Corporation, 1969* 

HOUGH, ROBBIN R* 

1968 "Casualty Rates and the War on Poverty*** American Economic Review , vol* 58* 

no* 2, pp* 528-532* 

HOVLAND, CARL I. 

1959 "Reconciling Conflicting Results Derived from Experimental and Survey 

Studies of Attitude Change*" American Psychologist , vol* 14, no, 1, 
pp* 8*17* 

HUTCHINSON, GEORGE B* 

1960 "Evaluation of Preventive Services*" Journal of Chronic Diseases , 

vol* 11/ no* 5, pp* 497*508* 



51 



HYMAN, HERBERT H* # and CHARLES R. WRIGHT 

''Evaluating Social Action Programs*** In Paul F* Lazarsfeld^ William h* 
Sewell^ and Harold L* Wilensky (eds*) The Uses of Sociology . New York; 
Basic Bobks^ Inc*^ Publishers^ pp* 741-782* 

JAMES # GEORGE 

1958 "Research by Local Health Departments: Problems^ Methods^ Results*** 
American Journal of Public Health * vol* 48^ no* 3^ pp* 353-361* 

JENKS# CHARLES L* 

1967 "Evaluation for a Small District**' Educational Product Report ^ vol* 2# 

no* 5# pp* 8-17* 

JONES, JAMES A. ^ 
1909 "Research*" In Breakthrough for Disadvantaged Youth * Washington, D.C*: ^ 
U*S* Department of Labor, Manpower Administration, pp* 235*250* 

JUSTMAN, JOSEPH 

1968 "Problems of Researchers in Large School Systems**' Educational Forum , 

vol* 32, no* 4, pp* 429-437* 

KANDEL, DENISE B* , and RICHARD H* WILLIAMS 

1964 Psychiatric Rehabilitation s Some Problems of Research * New York: 
Atherton Press* Inc* 

KEIMAN, HOWARD R* » and JACK ELIN50N 

1968 "Strategy and Tactics of Evaluating a Large Scale Medical Care Program*** 
Proceedings of the Social Statistics Section * Washington, D*C* i 
American Statistical Association, pp* ]f69*-191* 

KLINEBERG , OTTO 

1955 "The Problem of Evaluation Research*" International Social Science Bulletin , 
vol. 7, no* 3, pp* 347-351* 



ERIC 



52 



KOGANr LEONARD Sw and ANN W. SHYNE 

1966 "Tender-Minded and Tough-Minded Approaches in Evaluative Research." 
Welfare in Review ^ vol. 4, no. 2, pp. 12-17. 

KRAUSEr ELLIOTT A. 

1966 "After the Rehabilitation Center." Social Problems ^ vol. 14, no. 2, pp. 197-206. 
LASORTEr kiCHAEL A. 

1968 "The Caseworker as Research Interviewer." American Sociologist r vol. 3, 
no. 3r pp. 222-225. - f. 

LEMKAUr PAUL V. , and BENJAMIN PASAMANICK 

1957 "Problems in Evaluation of Mental Health Programs." American journal of 
Orthopsychiatry , .vol. 27 , no. 1, pp. 55-58. 

CO 

LEMPERTr RICHARD ^ 

1966 "Strategies of Research Design in the Legal Impact study." Law and Society 

Review f vol. 1, rio. L, pp. 111-132. 

LERMANr PAUL 

1968 "Evaluative Studies of Institutions for Delinquents: Implications for 

Research and Social Policy." Social Work r vol. 12 , no. 4, pp. 55-64. 

LEVINE, ABRAHAM S. 

1967 "Evaluating Program Effectiveness and Efficiency* Rationale and Description 

of Research in progress." Welfare in Review ^ vol. S, no. 2, pp. i-ii. 

LEVINEr ROBERT A. 

1969 "Evaluating the War on Poverty." in james L. Sundquist (ed.) On Fighting 

poverty ; Perspectives from Experiences . New York: Basic Books , mc. 
Publishers r pp. 188-216. 



ERIC 



53 



LEVINSON* PERRY 

1966 ''Evaluation of Social Welfare Programs! Two Research Models." Welfare 

in Review * vol . 4 , no . 10, pp . 5^12 . 

LEVITAN, SAR A. 

1969 "Facts, Fancies and Freeloaders in Evaluating Anti-Poverty Programs." 
Poverty and Human Resources Abstracts * vol. 4, no. 6, pp. 13-16. 

LIKERT* RENSIS, and RONALD LIPPITT 

1953 "The Utilization of social Science*" in Leon Festinger and Daniel Katz (eds.) 
Research Methods in the Behavioral Sciences * New Yorki Holt, Rinehart & 
Winston, inc. , pp. 581-646. 

LINDVAUi, CM., and RICHARD C. COX ^ 
1971 Evaluation as a Tool in Curriculum Developments The IPI Evaluation ^ 
Program . AERA Monograph Series on Curriculum Evaluation, no, 5. 
Chicagoi Rand McNally & Co., 1971. 

LIPPia?T, RONALD 

1965 "The Use of Social Research to Improve Social Practice." American Journal 
of Orthopsychiatry , vol. 35, no. 3, pp. 663*-669. 

PUCHTERaAND* EIMER 

1967 "Research and the Dilemnas in Developing Social Programs." in Paul F. 

Lazarsfeld, William H. Sewell* and Harold Wilensky (eds.) The Uses 
of Sociology . New Yorks Basic Books, inc.. Publishers, chapter 18. 

MANGUH, GAPTH L, 

1968 "Evaluating Manpower Programs.*' Monthly Labor Review , vol* 91* no* 2, 

pp. 2 1-22 i 



54 



MANN I JOHN 

1965 "The Outcome of Evaluative Research*" In Mann# Changing Human Behavior * 
New Yorki Charles Scribner's Sons, pp* 191-241* 

MARRISi PETERi and MARTIN REIN 

1969 "Research*" In Harris and Rein* Dilemnas of Social Reform; Poverty and 
Community Action in the United States * New York i Atherton Press# inc*# 
pp* 191-207* 

MAULDIN^ W« PARKER^ and JOHN A* ROSS 

" 1966 "Family Planning Experiments! A Review of Design*" Proceedings of the 
Social Statistics Section * Washington, D*C, t American Statistical 
Association f pp* 278**282* 

MCDILL, EDWARD L* , MARY S* MCDILL, and J* TIMOTHY SPREHE 

1969 Strategies for Success in Compensatory Educatlont An ?Vppraisal of 
Evaluation Research . Baltimore! The John Hopkins Press* 

MCINTYRE, ROBERT B* , and CALVIN C* NELSON 

1969 "Empirical Evaluation of Instructional Materials*" Educational Technology , 
vol* 9# no, 2# pp* 24-27* 

MERTON, ROBERT K. 

1964 "Role of the Intellectual in Public Bureaucracy*" In Merton, Social 
Theory and Social Structure * Glencoe# Illinois* The Free Press* 
pp* 207-224* 

MEYER, ALAN S*, and STANLEY K* BIGMAN 

1968 "Contextual Considerations in Evaluating Narcotic Addiction' Control Programs* 
Proceedings of the Social Statistics Section * Washington, D,C* i American 
Statistical Association, pp* i; >180* 



55 



1965 "The Study of Mans Evaluating, Action Programs.*' Trans-action , vol. 2 ,\ 

pp. 38-39. 

MOREHEAD, MILDRED A. 

1967 "The Medical Audit as an Operational Tool. ** American Journal of Public 

Health s vol. 57 no. 9. pp. 1643-1656. 

MOSSr L. ' 

1955 **The Evaluation of Fundamental Education." International Social Science 
Bulletin f vol. If no. 3r pp. 398-417. 

NAGPAUL. HANS 

1966 '*The Developnent of Social Research in an Ad Hoc Community Welfare Organi- 

zation. " Journal of Human Relations f vol. 14 r no. 4 r pp. 620-633. 

OTTr JACK M. 

1969 "Classification System for Decision Situations: An Aid to Educational 

Planning anci Evaluation.** Educational Technology ^ vol. 9r no. 2^ pp. 20-23, 

OWENS r THOMAS R. 

1968 "Suggested Tasks an<5 Roles of Evaluation i:5pecialists in Education." O 

Educational Technology r vol. 8r no. 22 ^ pp. 4-10. ^ 

PERRY, S.E.r and LYMAN WVNiJIE 

1959 *'Role Conflicts ^ole Definition* and Social Change in a Clinical Research 
Organization. " Social Forces f vol. 38 , no. Ir pp. 62-65. 

POPHAMr W. JAMES et alir 

1969 Instructional Objectives . AERA Monograph Series on Curriculum Evaluation r 
no. 3. Chicago: Rand McNally & Co. 



ERIC 



56 



PROVUSr MAIiCOLM 

1969 "Evaluation of Ongoing Programs in the Public School System*" In Ralph W* 

Tyler (ed*) Educational Evaluations Nfew Roles ^ New Means * 63th Yearbook 
of the National Societv for the Study of Education* Chicago: The 
Society^ pp- 242^283* 

RODHAN, HYHAN, and RALPH L* KOLODNY 

1965 ''Organizational strains in the Researcher-Practitioner Relationship*'* 

In Alvin Gouldner and S-M* Miller (eds*) Applied Sociology; Opportunities 
and Problems * New Yorki The F4ee Pressr pp* 93^5;13* 

ROSENBLATT, AARON 

1966 "The Practitioner *s Use and Evaluation of Research*" Social Work f vol* 11*^ 

no* 2, pp* 248-251* 

ROSENSHXNE, BARAK 

1970 "Evaluation of Classroom Instruction*" Review of Educa1:ional Research ^ 

vol* 40r no* 2r pp* 279-300* 

ROSSI, PKTER H* 

1966 "Boobytraps and Pitfalls in the Evaluation of social Action Programs*" 
Proceedings of the Social Statistics section * Washington^ D*C* t 
American Statistical Association, pp* 127-132* 

SADOFSKi\ STANLEY 

1966 "Utilization of Evaluation Resultsi Feedback into the Action Program*" 
In June L* shmelzer (ed*) Learning in Action * Washington^ D*C*i 
Government Printing Office^ pp* 22^36* 

SCANLON. R*G* ' 
1968 "Innovajiion Dissemination* " Pennsylvania school Journal s vol* 116 r 
pp* 37S-376* 



SCHULBERGr HERBERT C,, and FRANK BAKER 

X966 "Program Evaluation Models and the Implementation of Research Findings." 
American journal of Public Health y vol. 56 ^ no. 7r pp. 1246-1255. 

SCHWARTZ, RICHARD D. 

1961 ''Field Experimentation in Sociolegal Research." Journal of Legal Education ^ 
vol. 13 r no. 3r pp. 401-410. 

SCRIVEN, MICHAEL 

1967 **The Methodology of Evaluation." in Ralph W. Tyler, Robert M. Gagne^ and 
Michael Scriven (eds*) Perspectives of Curriculum Evaluation ^ no* 1* 
Chicagot Rand McNally St Co., pp. 39-83. 

SHELDON, ELEANOR B. , and HOWARD E. FREEMAN 

1970 "Notes on Social Indicatorst Promises and Potential.** Policy Sciences f 
vol. I, pp. 97-111. 

- cO 

SHERWOOD, CLARENCE C. ^ 
1967 "Issufts in Measuring Results of Action Programs." Welfare in Review , 
vol. 5, no.' 7, pp. 13-18. 

SLOCUM, W. L. ^ 

1956 "Sociological Research for Action Agenciest Some Guides and Hazards.'* 
Rural SocioXc^ y, vol. 21r no. 2, pp. 191-199. 

SMITH, BRUCE L.R. 

l^fifi The RAND Corporation . Cambridget Harvard University Press. 

SMITHr JOEL, Fm\NCIS M. SIMr and ROBERT C. BEALER 

1960 "Client S'tructure and the Research Process." In Richard n. Adams and 

Jack J. Preiss (eds.) Human Organization Research . Homewood, illinoisi 
Dorsey Pressr chapter 4. 



58 



SOMERS, GERALD G. 

1965 "Research Methodology in t^e Evaluation of Retraining Programmes.** 

Madison: University of Wisconsin, Industrial Relations Research Institute, 
reprint series, no. 61. Reprinted from Labour and Automation , Bixlletin 
No. 1, Geneva. 

STAKE, ROBERT E. ' 

1967 "The' Countenance of Educational Evaluation.** Teachers College Record s 

vol. 68# no. 7# pp. 523-540. 

STEIN, HERMAN D. 

1962 "The Study of Organizational Effectiveness.*' in David Fanshel (ed.) 

Research in Social Welfare Administration . New York: National Association 
of Social Workers, pp. 22-32. 

STEMBER, CfiARLES H. 

1968 **Evaluating Effects of an Integrated Classroom.** The Urban Review , vol. 2# 

no. 7# pp. 3-4# 30-31. 

STEWARD, M. A. 

1966 **The Role and Function of Educational Research - I.** Educational Research , 

vol. 9, no. 1 # pp. 3-6. 

STOUFFER. SAMUEL A. 

1950 "Some Observations on Study Design.** American Journal of Sociology , 
vol. 55 # no. 4, pp. 355-361. 

STtJFFLEBEAM, DANIEL L. 

1968 *'Evaluation as Enlightenment for Decision-making." Address delivered at 
the Working Conference on Assessment Theory, sponsored by the Commission 
on Assessment of Education Outcomes* The Association for Supervision and 
Curriculum Development, Sarasota* Florida, January 19. Columbus* The 
Evaluation Center, College of Education, The Ohio State University. 



59 



STURZ, HEr3ERT 

1967 "Experiments in the Criminal Justice System." Legal Aid Briefcase ^ pp. 1-5. 
SUCHMAN, EDWARD A. 

1970 ''Action for What? A Critique of Evaluative Research." In Richard O'Toole (ed.) 
The Orqanigation, Management and Tactics of Social Research . Cambridge t 
Schenkman Publishing Company. 

TAKISHITA, JOHN Y. 

1966 "Measuring the Effectiveness of a Family Planning Program* Taiwan's 

Experience. " Proceedings of the Social Statistics Section . Washington, D.C. : 
American Statistical Association, pp. 268-271* 

TAYLOR, PHILIP H. 

1966 '*The Role and Function of Educational Research." Educational Research , <Z> 

vol. 9, no. 1, pp. 11-15. . 

THERKILDSEN, PAUL, and PHILIP RENO 

1968 *'Cost"Benef it Evaluation of the Bernalillo County Work Experience Project." 

Welfare in Review , vol. 6, no. 2, pp. 1-12. 

TURVEY, RALPH, and A.R. PREST 

1965 *'Cost"Benef it Analysis t A Survey." Economic Journal , vol* 75, no. 300, 

pp. 683-735. 

TYLER, RALPH W. 

1966 "Assessing the Progress of Education." Science Educatio n, vol. 50, no. 3, 

pp. 239-242. 

U.S* DEPARTMENT OF HEALTH, EDUCATION AND WELFAEUE, NATIONAL ADVISORY MENTAL 
HEAI/TH COUNCIL 

1955 Evaluation in Mental Health . Washington, D.C. t Public Health Service, 
Publication no. 413. 



60 



U*S* DEPARTMENT OF HEAI/TH. EDUCATION AND WELFARE. OFFICE OF EDUCATION 

1970 Prej^arinq Evaluation Reports: A Guide for Authors * Washington. D-C* : 
Government Printing Office* 

U.S* HOUSE OF REPRESENTATIVES, COMMITTEE ON GOVERNMENT OPERATIONS^ RESEARCH AND 
TECHNICAL PROGRAMS SUBCOMMITTEE 

1967 The Use of Social Research in Federal Dotnestfic Programs * 90th Congress, 
1st session* Washington^ d*C*: G^- rnment Printing Officer vol* 3* 

WALL, w.D. 

1968 "The Future of Educational Research*" Educational Research *- vol. 10 r 

no* 3, pp* 163-169* 

WARD, DAVID A*, and GENE G* KASS&BAUM 

1967 "Evaluations of Correctional Treatmentr Some Implications of Negative 

Findings*" In S*A* Yefsky (ed*) Law Enforcement Science and Technology * 
Proceedings of the First National Symposivmi oi^ Law Enforcement Science 
and Technology^ llT Research institute* Washington^ D*C*i Thompson 
Book Co* r pp* 201-209* ^ 

WARDROP, JAMES L. 

1969 "Generalizability of Program Evaluation; The Danger of Limits*" Educational 

Product Report s vol* Zr no* 5r pp* 41-42* 

WEINBERGER^ MARTIN 

1969 "Evaluating Educational Programs: Observations by a Market Researcher," 
Urban Review , vol* 3r no* 4, pp* 23-26* 

WEISS, CAROL H. 

1967 "Evaluation of, In-Service Training*" In Targets for In-Service Training * 

Washington^ D*C* r Joint Commission on Correctional Manpower and Training^ 
pp* 47-54* 



61 



WEISSi SOBER!r S., and MARTIN REIN 

1969 "The Evaluation of Broad-Aim Programs; A Cautionary Case and a Moral*'* 
The Annals of the American Academy of Political and Social Science j 
vol* 385# pp. 118-132* A revised version also appears , in Administrative 
Science Quarterly , vol* 15# no* 1 (1970), pp* 97-109* 

WELCH, WAYNE V7* 

1969 "Curriculuun Evaluation*" Review of Educational Research i vol* 39# no; 4, 

pp* 429-443* 

WELTY, GORDON A* 

19S9 "Experimental Designs and Applied Research*" California Journal of 
Educational Research i vol* 20, no* 1/ pp* 40^-44* 

WESTBURy^ IAN 

1970 "Curriculum Evaluation*" Review of Educational Research , vol* 40, no* 2, 

pp* 239-260* C<1 

WHOLEY, JOSEPH S*, et al* 

1970 FederaX Evaluation Policy * Washington, D*C*j *^he Urban Institute* 

WILDER, DAVID E* 

1959 **Probleins of Evaluation Research*'* In Edmund DeS* Brunner, David E* Wilder, 

Corinne Kirchner, and John S, Newberry (eds*) An Overview of Adult Education * 
Chicago: Adult Education Association, pp* 243-273* 



WILKINS, LESLIE T* 

1964 "Evaluation Of Training Programs*" in Social Deviance * London* Tavistock 
Publications * pp* 288-293* 



62 



WIIJ>IAMSr WAI/TER 

1969 "Developing an Agency Evaluation Strategy for Social Action Programs." 
Journal of Human Resources , vol. 4, no. 4, pp. 451-465. 

WITTROCK, M.C. 

1969 "The Evaluation of instruction t Cause and Effect Relations in Naturalistic 
Data." Evaluation Comment ( Journal of the Center for the study of Evalua- 
tion , UCLA), vol. 1, no. 4r pp. 1-7. 

WRIGHT, CHARLES R. , and HERBERT H. HVMAN 

1964 "The Evaluators." m Phillip E. Hammond (ed.) Sociologists at Work t 

Essays on the Cr aft of social Research . New ^orkt Basic Books, inc.. 
Publishers, pp. 121-141 

WORTHEN, BLAINE R. 

1968 "Toward a Taxonomy of Evaluation Designs." x.^ucational Technology , vol. 8, 
no. 15, pp. 3-9. 



CO 



!ERJC 



Appendix C 



63 



EVALUATION FORM 
Presentation Instruction 
(Lecture, Demonstration, or Conference) 



Instructor 


Class 


Time 


Date 


Lesson Title 


Lesson Type 


Room 


School 



Instructions: Rate each of the Personal and Instructional Qualities on a 
scale of 1-5, with "1" as high and "5" as low* 



RATINGS 



COMMENTS 



Personal Qualities 

1* Appearance and Bearing 

2* Voice 

3 * Speech 

4* Platform Manner 

5* Teaching Personality 

Instructional Qualities 
6* Knowledge of subject 

7* Preparation and Planning 
8* Questioning Technique 
9* Student Participation 
10* Selection and Use of 
Training Aids 

11* Lesson Introduction 

12* Lesson Development 

13* Lesson Summary 

14* Management 

15* Control and Discipline 

16* Achievement of Objective 



Rater's Signature 



